Dataiku solutions By Dheerendra Yadav

  If you are working  on Spark SQL and you want to extract previous month last date from existing date  and unfortunately your present date in  text format e.g '20230301' Problem : You have column called "process_date" which contain date like '20230310' (March) and in same table you you have another date in same column have date like '20230228' (Feb) which is the last date of previous month.   Existing date -  '20230310'   Requirement :- To extract '20230228' (Last date of previous month) select     process_date   from table1    where  process_date =                   (                          select max(process_date)                                            ...

 If you have date in string datatype like '20230301' and  you want to convert it into 2023-03-01 in data datatype then please find below examples. Solution 1 select process_date, to_date(process_date, 'yyyyMMdd' ) as new_date from table Now you will below results: process_date new_date 20230603             2023-06-03 Solution 2 If you want to get some prior date on base of exiting date and you date in string datatype 'YYYYMMDD' e.g 20230603             select process_date, ,date_add(to_date(process_date, 'yyyyMMdd' ), - 1 ) as new_date from table Results: process_date new_date 20230603             2023-06-02  

 Use Alter command to add new column with CASCADE. (1) alter table `db_name`.`table_name` add columns(flag string) CASCADE; (2)  Please assign database name to both source and target tables to sync metastore with DB. (3) Add column in schema with partition table. (4) Then execute below Hive recipe. insert overwrite table `db_name`.`emp_part1` partition(job) SELECT  `empno` ,`ename` ,`hiredate` ,`salary` ,`deptno` ,flag ,`job`   FROM `prd_tbl_uacolob_hk_nsen`.`Emp_3`

 import os import dataiku import sys import time client = dataiku.api_client() project = client.get_project('project_id') try:          if len(os.listdir('/path/incoming'))==12:         scenario = project.get_scenario('Scenario_ID')         scenario.run_and_wait() except:     sys.exit(0)

Global Variables: {   "v_filename": "sas1234",   "v_file": "sas1234.sas7bdat",   "v_path": "/path_smptp/",   "v_file_date": "202301" }   Scenario (Python code) : import os import dataiku import sys import time import glob client = dataiku.api_client() project = client.get_project('MIGRATION') scenario = project.get_scenario("MIgration_FILES") os.chdir('/smtp_path') filepath='/smtp_path/' for file in (glob.glob('*.sas7bdat')):     #file='sas1234.sas7bdat'     filenm=file.split('.')[0]     partdt=filenm.split('_')[-1:][0]     filenm=filenm.replace('_'+partdt,'').strip()          project_variables = project.get_variables()     project_variables['standard']['v_filename']=filenm     project_variables['standard']['v_file']=file     project_variables['standard']['v_path']=filepath     p...

 import dataiku import pandas as pd d_dataset=dataset("abc") d_df=d_dataset.get_dataframe() d_df.columns = d_df.columns .str.replace("[$_() ]".'') df_output=dataiku.Dataset("df_output") df_output=write_with_schema(df_output) Note:- Dataframe name should be same when you apply replace function as amrked in yellow.