Crossformer

Abstract

1. Introduction

This is the origin Pytorch implementation of Crossformer: Transformer Utilizing Cross-Dimension Dependency for Multivariate Time Series Forecasting.

Figure 1. DSW embedding. Left: Embedding method of previous Transformer-based model: data points in different dimensions at the same step are embedded into a vector; Right: DSW embedding of Crossformer: in each dimension, nearby points over time form a segment for embedding.

Figure 2. TSA layer. Left: Overall structure: the 2D vector array goes through the Cross-Time Stage and Cross-Dimension Stage to get corresponding dependency; Middle: Directly using MSA in Cross-Dimension Stage to build the $D$-to-$D$ connection results in $O(D^2)$ complexity. Right: Router mechanism for Cross-Dimension Stage: a small fixed number ($c$) of routers'' gather and distribute the information among dimensions. The complexity is reduced to $O(2cD) = O(D)$.

2. Objective

Official implementation of our ICLR 2023 paper "Crossformer: Transformer Utilizing Cross-Dimension Dependency for Multivariate Time Series Forecasting"

This project demonstrates how Python, Deep Learning can be applied to a real-world Data Science problem.

4. Technology Stack

5. System Requirements

General requirements for this technology stack — check the README for exact versions.

• Python 3.8 or later
• pip / virtualenv for dependencies
• VS Code, PyCharm or Jupyter Notebook
• Git (to clone the repository)

6. Installation & Setup

git clone https://github.com/Thinklab-SJTU/Crossformer.git
cd Crossformer

1. This is an hourly-sampled dataset with 13 dimensions. And we are going to use the past week (168 hours) to forecast the next day (24 hour) and the segment length is set to 6. Therefore, we need to run:
2. We can evaluate the trained model by running:

python main_crossformer.py --data AirQuality --data_path AirQualityUCI.csv --data_dim 13 --in_len 168 --out_len 24 --seg_len 6

python eval_crossformer.py --setting_name Crossformer_AirQuality_il168_ol24_sl6_win2_fa10_dm256_nh4_el3_itr0 --save_pred

Full setup instructions are in the project README.

7. Future Enhancements

Suggested extensions you can add to make this your own project.

• Turn the analysis into an interactive dashboard
• Automate data refresh with a scheduled job
• Add a predictive model on top of the analysis

8. Viva / Review Questions

Common questions examiners ask for projects in this domain.

1. What is the source of the dataset and how was missing data handled?
2. Which exploratory analysis steps revealed the most useful insight?
3. Why were these particular charts chosen to present the data?
4. Which statistical or ML technique supports the conclusions?
5. How could the analysis be automated or refreshed with new data?

9. Source Code & License

This project is developed by Thinklab-SJTU and published on GitHub under the Apache License 2.0. Please follow the license terms and credit the original author when you use or modify this code.