Define variable in scenario

 If you want to declare variables in dataiku in scenario and access them in project.


There is sequence in dataiku to declare variable.


(1) Define variable :- Where you can build logic or condition to create variable

      Example 

                 v1=100

(2) Set project variable steps:  Assign v1 to another variable v2.

      example :

                v2-v1

(3) Execute and check v2 will automatically define Global variables.

          

If you want to take already define variable in Global variable and take as parameter in "Define variable" 


var1  ----> variables["dt_param"]


(4)

 If you have specific date and you want to get other date bases on on your given date in scenario then "Go to - Define Variable " create variable and use below code

var1-> inc(asDate(variables["dt_param"],'yyyy-MM-dd'),toNumber(concat('-',datePart(asDate(variables["dt_param"],'yyyy-MM-dd'),'Day'))),'day')

Comments

Post a Comment

Popular posts from this blog

Date format issue with spark sql

 If you have date format like  date_col 1/1/1978 0:00 6/6/1975 0:00 1/1/1978 0:00 1/1/1978 0:00 7/23/1988 0:00 1/1/1978 0:00 1/26/1961 0:00 3/3/1980 0:00 6/2/1984 0:00 1/2/1971 0:00 Solution : substring(TO_DATE(date_col ,'MM/dd/yy'),0,10) from table Result : 1978-01-01 1975-06-06 .. 1988-07-23 ..
Read more

Hive Partition sub folders HIVE_UNION_SUBDIR_1,HIVE_UNION_SUBDIR_2,HIVE_UNION_SUBDIR_8

 Hive Partition have sub folders like HIVE_UNION_SUBDIR_1,HIVE_UNION_SUBDIR_2,HIVE_UNION_SUBDIR_3 Problem : When you use UNION ALL in query with Hive version 1.2.0 onward. UNION ALL will not supported and we have TEZ engine setup in hive-site.xml or hive configuration file which is responsible to create  sub folders like HIVE_UNION_SUBDIR_1 etc  on HDFS. When you use spark sql or other sql query to read Partition data then your resultant partition created with blank or 0 value. Solution: Please change the configuration file of hive and set below property. Its work for me in dataiku. hive.execution.engine=mr It will invoke Map Reduce process which will  bit slow to process the job but it will help to stop creating the extra sub folder on HDFS.  
Read more

Dataiku and Dremio date difference

In dataiku you have applied to_date function to convert string to date datatype in SparkSQL and when query same dataset in dremio then you will difference of 1 day. Example -  In dataiku      SELECT  offer_date  FROM temp      Result :      2023-05-18T00:00:00.000Z Dremio :     SELECT  offer_date  FROM temp Result :      2023-05-17 08:00:00.000 Now you will find the difference of  1 day and 8 hours (sometime 16 hours)  between Dataiku and Dremio. Solutions:       This difference is due to time difference. In Dataiku we have  correct date to correct it Dremio please use TO_TIMESTAMP function to get correct date. Example : select TO_TIMESTAMP(offer_date, 'yyyy-mm-dd' ) as offer_date from temp
Read more