Time Series Forecasting Test Issue

[leonardo95]

Hi,

I have a dataset with three years of data. To have more data, I split the day into 12 hours.
I have been trying to predict the frequency in the next 365 days. However, I received the error "ValueError: Initial pipeline fit was failed due to: There are no common indices for outputs. Check pipeline structure and the correctness of the data"

Here is my code:
tickets_count_afa5c858-4907-4c5d-b6fa-3b791b0d2eb7_year_3.csv

    dataset = pd.read_csv(dataset_path, header=0, sep=";", low_memory=False)
    task = Task(TaskTypesEnum.ts_forecasting, TsForecastingParams(forecast_length=720))
    train_input = InputData.from_csv_time_series(file_path=dataset_path, task=task, target_column=target_column, delimiter=";")
    train_data, test_data = train_test_data_setup(train_input)

   model = Fedot(problem='ts_forecasting', task_params= task.task_params, timeout=1, cv_folds= 2)
   pipeline = model.fit(train_input)

    out_sample_forecast = model.forecast(predict_data)
    print("First out:", out_sample_forecast)
    print('Metrics for out-sample forecast:',
          model.get_metrics(metric_names=['rmse', 'mae', 'mape']))
    model.plot_prediction()

The dataset is attached.
Can someone help me?
tickets_count_afa5c858-4907-4c5d-b6fa-3b791b0d2eb7_year_3.csv

[Lopa10ko]

Hi, @leonardo95
Thanks for reaching out!

It seems, that you are following an example of a time series forecasting pipeline initiation.
Our team added #1241 rather recently, so we would recommend cloning master branch of FEDOT and working within FEDOT project until these changes are included in the new release.

Also there is no predict_data initialization in your code snippet, so here is a listing of the working code:

dataset_path = ...
target_column = 'frequency'
dataset = pd.read_csv(dataset_path, header=0, sep=";", low_memory=False)

task = Task(TaskTypesEnum.ts_forecasting, TsForecastingParams(forecast_length=720))
train_input = InputData.from_csv_time_series(file_path=dataset_path, task=task, target_column=target_column, delimiter=";")
train_data, test_data = train_test_data_setup(train_input)

model = Fedot(problem='ts_forecasting', task_params=task.task_params, timeout=1, cv_folds=2)
pipeline = model.fit(train_data)

out_sample_forecast = model.forecast(test_data)
print("First out:", out_sample_forecast)
print('Metrics for out-sample forecast:',
      model.get_metrics(metric_names=['rmse', 'mae', 'mape']))
model.plot_prediction()

Output on your data

First out: [  35.8554437    48.97811327   49.97771604   33.63622985   39.75589645
   38.14596527   39.90041377   51.06376194   51.19028739   33.06769749
   27.38357762   25.09148445   31.03462989   22.66435027   47.53018189
   34.21412776   34.39478027   24.91817331   32.24934916   28.03327787
   34.45185898   43.011668     42.07163461   50.24587885   36.84998969
   40.50641548   21.8377169    35.7523156    38.55448376   48.07064982
   46.77577472   46.78897083   39.48382913   37.08313979   46.08681009
   37.11833009   51.07074359   43.56420324   42.6891434    56.85407492
   53.50752098   51.77979773   54.67549298   49.56655148   59.52762911
   53.22229006   58.95513139   48.62651445   53.50346154   36.48073782
   51.23774184   45.94435324   40.87812688   51.08618247   44.00622693
   39.20773366   33.01904392   47.8282285    29.86988717   45.986217
   46.70554497   37.00299849   37.28550197   31.8125694    42.04348707
   40.2018435    40.34299116   43.95497017   42.39236432   26.83572663
   36.88787202   41.74400116   34.4564917    47.70480518   38.94121634
   42.67368136   41.21863896   41.83748634   31.96654607   54.68036794
   41.368401     47.21434436   34.86126947   31.13847518   35.14069573
   38.17975034   30.25852525   37.16195822   37.75257153   32.6170361
   38.43090646   43.00361831   40.34609298   36.45015346   34.74401365
   33.51661054   54.24726985   43.54980525   39.10856796   36.64956652
   37.43797968   35.72638771   41.34253282   38.05955699   45.51553008
   42.66509171   38.98923718   42.27558486   44.69630847   39.84944309
   53.06942073   52.43782341   54.09998675   56.17796615   41.60628798
   36.94391483   43.86030783   51.20836576   46.40649923   43.41060936
   53.58988329   43.59816424   51.26552915   31.29154564   49.34694588
   41.00586706   44.49985905   43.72307856   47.97039393   36.60608119
   43.12808021   38.42811952   34.42442794   42.66859285   39.19569948
   39.09116839   29.74245609   37.30120183   29.79654243   25.48476608
   26.30844429   44.06323898   32.22719864   44.819685     37.64934691
   35.61029838   28.0785184    37.58090079   37.06490576   43.92341433
   54.5586815    42.77263202   36.50852216   34.09180521   28.98000133
   46.03305514   45.97023993   52.55173707   55.21169009   49.75517632
   46.45175178   43.45693744   52.78636136   55.28577465   54.36261295
   49.6927105    49.19169031   47.50665369   45.06577538   49.79880932
   42.47944836   45.46125357   46.12892847   45.07857103   51.57614437
   47.29663227   39.72596387   36.47657553   48.18082776   47.48920811
   48.54188281   35.4256829    42.18575748   32.73412208   30.8487981
   38.16421441   39.92446124   40.08525369   36.9416415    45.35788557
   33.81924347   40.55082096   31.90807254   29.47610736   36.49164211
   42.95661576   35.39764655   40.12045225   40.88932013   42.56369756
   38.40927522   36.95835009   41.55292151   51.34758582   42.6011074
   44.60885198   40.23172362   41.58741986   47.67853845   40.55730546
   37.29123709   32.39858499   39.56034102   38.96204038   42.53058973
   42.43632792   38.62883107   45.88424009   41.26620052   38.14266152
   50.49138322   43.13341739   33.98565366   35.86171259   46.46951235
   35.95912892   36.25729167   38.77338458   42.54571355   42.99282375
   45.80661186   45.48193363   42.75608904   53.63552284   50.64888748
   46.29111341   44.11236218   47.85312429   43.13023932   43.72591316
   39.2806495    37.57717242   37.39933703   43.72200308   47.26698254
   42.75943076   48.50588005   46.03303559   42.44637243   43.57914552
   46.81875153   35.54530396   42.02581351   39.76176205   40.6562083
   36.1921512    38.77871236   33.54005359   38.16155981   32.6810849
   37.5226152    38.27247972   39.00916741   44.30769657   33.82089253
   28.86656035   34.31562297   37.67131196   29.6653985    40.61182588
   37.81060564   31.70649687   32.31273104   44.87546761   41.03701482
   44.04411221   33.66764432   47.95295222   51.25765727   47.92915407
   40.89953809   44.24604844   44.6965754    51.44317323   57.34691312
   45.40753536   50.75760627   47.0709892    44.07436979   52.4873988
   62.35654688   52.14177053   49.94393734   49.46863669   47.6707212
   47.47927986   42.99864911   46.52701493   52.41720845   48.59851854
   42.70601868   44.93538643   47.58569774   40.89428975   48.93586616
   44.17055804   47.9302688    51.45419674   26.05080843   33.94969052
   35.43307801   35.68051868   34.37761855   35.17654361   37.05242502
   36.63357223   25.11761066   31.87340531   36.04544258   42.46957357
   31.80424545   35.72737001   33.07364893   40.71298658   26.25924911
   32.11020063   40.62151094   49.64682399   40.10650395   36.90361312
   43.77744148   37.61644751   38.10460519   40.87433989   46.13772567
   46.32011225   40.90891962   36.70106444   37.79066643   43.23438917
   38.2252054    37.70178673   40.78013922   50.46388915   56.78587762
   48.93150959   55.86867668   51.91464094   53.90112963   53.97466246
   62.39327442   56.69450997   53.52033107   48.23408032   48.14862872
   41.82187242   51.7700077    40.91569978   43.9142875    49.15763348
   44.91325966   38.58225561   36.40049217   30.27566511   48.72528919
   41.51421323   45.30738753   38.54565998   36.61983004   30.53896159
   45.62181119   46.2845115    45.39279193   36.20473189   36.57036338
   39.15523123   43.58625701   41.29632518   33.25203479   32.7111444
   39.62756672   40.58290571   43.50385374   46.09081674   43.86062255
   33.04997267   29.32371414   39.29111102   44.30397777   36.90707349
   33.71196228   38.71953538   38.86660217   33.48921279   32.47798049
   40.47985363   37.7182459    46.74818849   41.95421332   41.42339053
   40.17325023   43.77259808   41.22334324   46.16714087   43.37753714
   43.99171242   39.80255951   37.19587213   38.71014907   42.84538987
   36.58636882   44.36690048   42.80264597   55.20444432   41.22933279
   49.74705724   45.31304984   44.54283631   43.80092005   55.83166195
   46.41876825   48.68913233   53.89534162   48.86754761   41.87248875
   38.89733175   44.28419678   50.31543146   46.7764217    48.04446017
   44.91501307   34.31035707   35.25262965   36.8431992    44.73719325
   44.58238085   50.97098968   35.15982911   30.44674118   29.02076381
   35.16835521   33.3823347    39.06613195   30.08022374   30.92049518
   25.74085017   32.53057091   34.42172225   46.33309347   35.82327663
   39.37894162   37.15708235   44.0831285    36.70467108   37.10767572
   39.32666686   31.34644086   47.2140047    47.34585587   48.47580435
   46.2162373    43.36295634   41.54087189   47.22043831   50.92218875
   53.12928431   49.65573623   43.53848478   47.73419842   52.70213915
   48.32670515   59.72695974   50.34121191   57.67326742   53.16996427
   57.1683738    46.25803451   46.56317072   44.07531763   45.54810543
   54.94495142   45.45341146   39.06602113   35.46789647   42.40416013
   41.28663868   43.24995308   42.33242572   36.209265     40.32574716
   42.44024538   35.55012425   40.47791233   36.23419209   35.7840143
   38.96623446   38.21290739   37.69666612   35.73872834   31.4718977
   38.33874712   38.46991258   37.54282731   41.04362604   41.71899819
   41.87439349   45.05879368   42.73343052   39.02013876   39.36019379
   43.13029288   37.56659133   41.07781368   38.55819479   33.85742827
   41.59650975   45.67587665   39.39713024   36.55648403   37.74823132
   40.19242923   44.25442078   39.81904821   45.83999805   40.37268047
   40.84753573   35.30308574   46.40522956   40.81941678   38.72541443
   39.44755543   36.21804111   46.94058239   42.85408328   36.91898923
   43.89649686   41.33678792   50.29391125   43.12606113   46.12991252
   42.47906175   50.88764513   40.56592282   43.06340508   54.36806801
   41.74567153   46.43349079   45.64544605   42.55048313   40.85926417
   42.21937039   46.14024118   49.51719471   39.01553867   43.09389663
   43.31553286   40.386735     40.66097351   48.74442967   38.00623393
   39.779322     35.18367822   40.95042158   32.60442013   44.83768996
   29.29551091   36.07808563   32.29056325   34.31066614   33.31430637
   35.05824669   31.73924035   37.67267624   38.51171389   34.81868376
   39.71045414   34.43398714   39.55412034   42.74119718   40.8795764
   36.38899323   48.32338925   33.70682213   47.44227385   42.75779638
   45.67149837   48.1928427    50.41779185   46.89389198   49.1202342
   49.70680812   47.39090356   50.81214438   50.35357938   51.22073198
   50.21114813   43.65297733   51.0044684    51.44118986   49.23331986
   46.58087557   50.84191417   41.57042377   45.0519989    50.53238442
   42.55080428   37.51536741   35.44874566   44.92901265   51.80795936
   35.06225566   42.74759472   32.89812549   36.18139855   40.09575782
   43.91320512   42.22932926   28.99478296   40.18300631   23.20735001
   35.08699937   33.53856103   40.31037232   25.38921173   40.5951971
   38.37478152   38.01088748   30.1446577    44.51254674   41.49853977
   43.7680788    51.09482474   47.22569175   38.69397724   44.27682098
   37.17319889   47.26022419   44.16318119   48.20805713   40.4189552
   39.60126022   47.28618498   42.1684149    41.70321713   42.95640361
   55.32103806   46.48495645   40.63680591   42.8406556    30.09682559
   39.83631666   41.66451542   39.10797062   35.81860282   48.71863263
   39.69247709   27.36739625   52.9224333    34.90574404   57.01032748
   37.15199552   49.754964     45.41255076   44.20823288   31.13231818
   44.65171966   52.67973859   38.52389079   45.14114171   40.78766151
   35.29368472   45.01940731   38.21302168   45.2895017    37.88183741
   55.09084975   39.72142652   40.64539639   32.75646501   38.44318411
   44.77234178   35.25577811   54.81504818   41.51074151   37.59223967
   41.05138248   43.35657771   36.57019663   43.47198823   35.30155943
   33.80905484   37.87273295   37.63058218   34.62932066   30.19208267
   30.13804052   39.62192132   34.83433888   52.15091248   45.44806082
   32.14964612   32.66169868   53.19736499   38.44260025   43.0043196
   44.08462517   46.69164406   44.77730507   44.88079797   33.42748579
   46.04406403   56.08669335   57.54064696   64.18670601   64.20574056
   58.74244687   45.48588472   50.87967762   59.338244   -175.10532967]
Metrics for out-sample forecast: {'rmse': 29.097, 'mae': 26.025, 'mape': 1.729}

Let us know if the problem has been solved or if there are any difficulties

[leonardo95]

Hi,

I managed to work but i noticed something in your code. Why using the model fit with the train_input and not train_data? I believe this was the issue during my experiments.

Thank you!

[leonardo95]

Hi again,

I tried using the train_data instead of all data (to avoid overfitting) but a valuerror is raised "ValueError: Initial pipeline fit was failed due to: There are no common indices for outputs. Check pipeline structure and the correctness of the data". I suspect this may be happening due to some behavior in the data I shared previously.

I have been using FEDOT for a few months now and I'm happy with the results, excluding this case where I don't comprehend the problem.

Appreciate the help!

[Lopa10ko]

I tried using the train_data instead of all data (to avoid overfitting) but a valuerror is raised "ValueError: Initial pipeline fit was failed due to: There are no common indices for outputs. Check pipeline structure and the correctness of the data".

There seems to be a problem for high values of parameter forecast_length for a dataframe of 1400 records total.

So here are the possible solutions:

Code listing for in-sample forecast

task = Task(TaskTypesEnum.ts_forecasting, TsForecastingParams(forecast_length=180))
train_input = InputData.from_csv_time_series(file_path=dataset_path, task=task, target_column='frequency', delimiter=";")
train_data, test_data = train_test_data_setup(train_input, validation_blocks=4)

model = Fedot(problem='ts_forecasting', task_params=task.task_params, timeout=1, cv_folds=2)
pipeline = model.fit(train_data)

# len(test_data.target) = forecast_length * validation_blocks
in_sample_forecast = model.predict(test_data, validation_blocks=4)

print("First out:", in_sample_forecast)
print('Metrics for in-sample forecast:',
      model.get_metrics(metric_names=['rmse', 'mae', 'mape'], target=test_data.target, validation_blocks=4))

Code listing for out-sample forecast

task = Task(TaskTypesEnum.ts_forecasting, TsForecastingParams(forecast_length=1))
train_input = InputData.from_csv_time_series(file_path=dataset_path, task=task, target_column='frequency', delimiter=";")
train_data, test_data = train_test_data_setup(train_input)

model = Fedot(problem='ts_forecasting', task_params=task.task_params, timeout=1, cv_folds=2)
pipeline = model.fit(train_data)
out_sample_forecast = model.forecast(test_data, horizon=720)

print("Forecast:", out_sample_forecast)

with plt.ion():
    model.plot_prediction()
    plt.savefig('tickets.pdf')
    plt.close()

[leonardo95]

Hi,

So, there is no way to test the model performance with a year of training data and another year as a test dataset? I was trying to analyze the model capacity in generalizing data for 1 year.

Thank you!

[Lopa10ko]

In a nutshell, FEDOT uses cross-validation, but then splitting data into train and test with a standart ratio of 0.8 and forecasting half of a records, validation block (and only one) would be in a train part (>720 records).
Your options here are as follows:

• use one of the solutions above for in-sample or out-sample forecast
• predict with smaller forecast_length
• get more data, so there are several validation blocks
• build fixed pipeline (even a zero-shot works)

[leonardo95]

Hi,

I see. Thank you for pointing out possible solutions. What do you mean by fixed pipelines?

[Lopa10ko]

What do you mean by fixed pipelines?

A pipeline where you can add specific nodes. Basically, it works as an initial assumption.
Check out any example in our test module, here is one of them:

Example of a specific pipeline build

def get_pipeline():
    lagged = PipelineNode('lagged')
    ridge = PipelineNode('ridge', nodes_from=[lagged])
    rfr_parallel = PipelineNode('rfr', nodes_from=[ridge])
    ridge_parallel = PipelineNode('ridge', nodes_from=[ridge])
    final = PipelineNode('ridge', nodes_from=[rfr_parallel, ridge_parallel])
    return Pipeline(final)