Empowering Earth System Science through Julia for Optimized Processing and Statistical Driver Attribution in Big Data
The availability of remote sensing information of the Earth has increased exponentially over the last decades. This trend is expected to continue as new remote sensing products from new satellite missions become available. To keep the pace with the amount of information gathered by the various sensors orbiting the Earth, it is essential to develop tools that allow scientists and students to easily manipulate and perform operations on spatio-temporal gridded data. Ideally, the new generation tools should be able to run, and interact with data on cloud platforms rather than individual computers. In this pilot, we developed a new Julia package as an entrance point to high-performance computing based on the YAXArrays.jl package, the YAXArraysToolbox package [@DOI:10.5281-zenodo.7989936]. The front-end of the YAXArraysToolbox package is divided in two modules, the first one (Basic Operations, Media 1) contains a set of basic operations to process and visualize large gridded data in a very efficient way. The second module (Spatio-Temporal Analyses, Media 2) contains a set of tools to perform data-driven attribution, e.g., using the space-for-time concept, and spatio-temporal data partitioning for cross-validation analyses based on [@DOI:10.1038-s41467-017-02810-8], [@DOI:10.1016-j.envsoft.2017.12.001], and [@DOI:10.1111-2041-210X.13650]. We anticipate that the package will be beneficial to various domains within the Earth system data science, and Julia user communities by providing a user-friendly environment. Advanced users will also benefit by taking advantage of the implemented analyses to perform data-driven attribution using machine learning techniques and semi-empirical modeling. We intend to further enhance the capabilities of the package by incorporating additional functions useful for statistical driver attribution in large datasets, including forward feature selection based on regularized regression.
