Dataiku solutions By Dheerendra Yadav

 If you are getting below error while reading SAS file using Python or Output display b'  value'    with rows [13:47:17] [INFO] [dku.utils] - return lib.map_infer(values, mapper, convert=convert) [13:47:17] [INFO] [dku.utils] - File "lib.pyx", line 2972, in pandas._libs.lib.map_infer [13:47:17] [INFO] [dku.utils] - File "<string>", line 15, in <lambda> [13:47:17] [INFO] [dku.utils] - UnicodeDecodeError: 'utf-8' codec can't decode byte 0x95 in position 20: invalid start byte  Solution: Please use below code to for resolution: import pandas as pd, numpy as np df = pd.read_sas('path//20240710.sas7bdat') for col in df.columns:     if df[col].dtype == 'object':         df[col] = df[col].apply(lambda x: x.decode('utf-8','ignore') if isinstance(x, bytes) else x) print(df)

 Hive Partition have sub folders like HIVE_UNION_SUBDIR_1,HIVE_UNION_SUBDIR_2,HIVE_UNION_SUBDIR_3 Problem : When you use UNION ALL in query with Hive version 1.2.0 onward. UNION ALL will not supported and we have TEZ engine setup in hive-site.xml or hive configuration file which is responsible to create  sub folders like HIVE_UNION_SUBDIR_1 etc  on HDFS. When you use spark sql or other sql query to read Partition data then your resultant partition created with blank or 0 value. Solution: Please change the configuration file of hive and set below property. Its work for me in dataiku. hive.execution.engine=mr It will invoke Map Reduce process which will  bit slow to process the job but it will help to stop creating the extra sub folder on HDFS.  

 If you have file_name.sas7bdat and you would like to convert into normal csv file without   UnicodeDecodeError character issue then below code. >>>>>>>>Python code>>>>>>>>>>>>>>  from dataiku import pandasutils as pdu df = pd.read_sas('/path/file_name.sas7bdat') for col in df.columns:     if df[col].dtype == 'object':            df[col] = df[col].apply(lambda x: x.decode('utf-8') if isinstance(x, bytes) else x) print(df) ======================================================================================== Try this if above one didn't work. import dataiku import pandas as pd, numpy as np from dataiku import pandasutils as pdu import os import glob df = pd.read_sas(dataiku.get_custom_variables()["v_sg_cust_path"]+'/'+dataiku.get_custom_variables()["v_file"]) df= df.apply(lambda x: x.decode() if isinstance(x, bytes) else x) # Write recipe outputs landing = datai...

 If  you want to setup auto generated mail in dataiku which send success or failure status before or after job submit. Then please follow below steps. Steps 1. Create scenario. 2. Setting -  Reports 3. Add Reports 4. Mail - Provide name relevant  to your mail. 5. Run Condition - On 6. Channel - mail(smpt)       outcome == 'SUCCESS' 7. Sender - SMPT server name provide by your team which configuring mail server.         example - alerts@abc.com 8.  Recipients - To whom you want to send mail ( name1@gmail.com) 9. Subject - If you want to change the subject you can change it. 10. Steps :- One on build (database) 11. Save  12 Run Scenario

Please find  below code to convert csv file to xml.  # -*- coding: utf-8 -*- import dataiku import pandas as pd, numpy as np from dataiku import pandasutils as pdu from lxml import etree as et Dept_3 = dataiku.Dataset("Dept_3") df = Dept_3.get_dataframe() df['deptno'] = df['deptno'].astype(str) root= et.Element('data') # iterate over rows and add to the tree for ix, row in df.iterrows():     item = et.SubElement(root, 'item', attrib=row.to_dict()); # get a handle on the folder to write xml_files = dataiku.Folder("xfile") with xml_files.get_writer("dept.xml") as w:     w.write(et.tostring(root, encoding='UTF-8', xml_declaration=False))

 If you want to declare variables in dataiku in scenario and access them in project. There is sequence in dataiku to declare variable. (1) Define variable :- Where you can build logic or condition to create variable       Example                   v1=100 (2) Set project variable steps:  Assign v1 to another variable v2.       example :                 v2-v1 (3) Execute and check v2 will automatically define Global variables.            If you want to take already define variable in Global variable and take as parameter in "Define variable"  var1  ----> variables["dt_param"] (4)  If you have specific date and you want to get other date bases on on your given date in scenario then "Go to - Define Variable " create variable and use below code var1-> inc(asDate(variables["dt_param"],'yyyy-MM-dd'),toNumber(co...

Case 1  If  you have any date and you want to calculate last month date of previous month in data dataiku function then use "inc" and "datePart" function. inc(dt,toNumber(concat('-',datePart(dt,'Day'))),'day')  Example : Parameter date - 2023-06-26 Result :-  2023-05-31 Case 2 If you have specific date and you want to get  last month date of  previous months or 2 previous month data then use below code inc(inc(asDate("2023-06-26","yyyy-MM-dd"),toNumber(concat('-',datePart(asDate("2023-06-26","yyyy-MM-dd"),'Day'))),'day'),-1,'months') Example : Parameter date - 2023-06-26 Result :-  2023-04-30