<jupyter_start><jupyter_text>dataset.py  Command to create an empty dataset in AutoML dataset<jupyter_code>! python dataset.py create_dataset --dataset_name train<jupyter_output>Dataset name: projects/401832599639/locations/us-central1/datasets/TBL7687283924107526144
Dataset id: TBL7687283924107526144
Dataset display name: train
Dataset metadata:
	stats_update_time {
}

Dataset example count: 0
Dataset create time:
	seconds: 1564496871
	nanos: 340824000<jupyter_text>Command to list of existing dataset info in AutoML dataset<jupyter_code>! python dataset.py list_datasets<jupyter_output><empty_output><jupyter_text>Command to import new data into an existing empty AutoML dataset<jupyter_code>! python dataset.py import_data --dataset_id TBL7687283924107526144 --path "bq://hackathon1-183523.demo3_v2.train"<jupyter_output>Processing import...
Data imported.<jupyter_text>Command to list of existing dataset info in AutoML, to make sure the import is successful<jupyter_code>! python dataset.py list_datasets<jupyter_output><empty_output><jupyter_text>Schema review<jupyter_code>! python dataset.py schema_review --dataset_name "projects/401832599639/locations/us-central1/datasets/TBL7687283924107526144"<jupyter_output>Feature name:start_month
Feature name:FLOAT64

Feature name:dayOfWeek
Feature name:CATEGORY

Feature name:start_hour
Feature name:FLOAT64

Feature name:pickup_longitude
Feature name:FLOAT64

Feature name:price_per_mile
Feature name:FLOAT64

Feature name:trip_miles
Feature name:FLOAT64

Feature name:dropoff_longitude
Feature name:FLOAT64

Feature name:dayOfYear
Feature name:FLOAT64

Feature name:trip_seconds
Feature name:FLOAT64

Feature name:pickup_latitude
Feature name:FLOAT64

Feature name:dropoff_latitude
Feature name:FLOAT64<jupyter_text>list table spec id of the dataset<jupyter_code>! python dataset.py list_table_specs --dataset_id TBL7687283924107526144<jupyter_output>List of table specs:
Table spec name: projects/401832599639/locations/us-central1/datasets/TBL7687283924107526144/tableSpecs/5409614800843964416
Table spec id: 5409614800843964416
Table spec time column spec id: 
Table spec row count: 27589
Table spec column count: 11<jupyter_text>list each column spec info in the table<jupyter_code>! python dataset.py list_column_specs --dataset_id TBL7687283924107526144 --table_spec_id 5409614800843964416<jupyter_output>List of column specs:
Column spec name: projects/401832599639/locations/us-central1/datasets/TBL7687283924107526144/tableSpecs/5409614800843964416/columnSpecs/3796398146431483904
Column spec id: 3796398146431483904
Column spec display name: trip_miles
Column spec data type: type_code: FLOAT64

Column spec name: projects/401832599639/locations/us-central1/datasets/TBL7687283924107526144/tableSpecs/5409614800843964416/columnSpecs/4949319651038330880
Column spec id: 4949319651038330880
Column spec display name: pickup_latitude
Column spec data type: type_code: FLOAT64

Column spec name: projects/401832599639/locations/us-central1/datasets/TBL7687283924107526144/tableSpecs/5409614800843964416/columnSpecs/914094384914366464
Column spec id: 914094384914366464
Column spec display name: pickup_longitude
Column spec data type: type_code: FLOAT64

Column spec name: projects/401832599639/locations/us-central1/datasets/TBL7687283924107526144/tableSpecs/5409614800843964416/columnSpecs/3219937394128[...]<jupyter_text>get dataset<jupyter_code>! python dataset.py get_dataset --dataset_id TBL7687283924107526144<jupyter_output>Dataset name: projects/401832599639/locations/us-central1/datasets/TBL7687283924107526144
Dataset id: TBL7687283924107526144
Dataset display name: train
Dataset metadata:
	primary_table_spec_id: "5409614800843964416"
stats_update_time {
  seconds: 1564496993
  nanos: 135000000
}

Dataset example count: 27589
Dataset create time:
	seconds: 1564496871
	nanos: 340824000<jupyter_text>get table spec info<jupyter_code>! python dataset.py get_table_spec --dataset_id TBL7687283924107526144 --table_spec_id 5409614800843964416<jupyter_output>Table spec name: projects/401832599639/locations/us-central1/datasets/TBL7687283924107526144/tableSpecs/5409614800843964416
Table spec id: 5409614800843964416
Table spec time column spec id: 
Table spec row count: 27589
Table spec column count: 11<jupyter_text>get column spec id<jupyter_code>! python dataset.py get_column_spec --dataset_id TBL7687283924107526144 --table_spec_id 5409614800843964416 --column_spec_id 1490555137217789952<jupyter_output>Column spec name: projects/401832599639/locations/us-central1/datasets/TBL7687283924107526144/tableSpecs/5409614800843964416/columnSpecs/1490555137217789952
Column spec id: 1490555137217789952
Column spec display name: price_per_mile
Column spec data type: type_code: FLOAT64

Column spec data stats: distinct_value_count: 3249
float64_stats {
  mean: 6.26339242877
  standard_deviation: 4.0244233649
  quantiles: 0.00172413793103
  quantiles: 4.16666666667
  quantiles: 5.38461538462
  quantiles: 6.875
  quantiles: 95.0
  histogram_buckets {
    min: -inf
    max: 9.50155172414
    count: 25103
  }
  histogram_buckets {
    min: 9.50155172414
    max: 19.0013793103
    count: 2239
  }
  histogram_buckets {
    min: 19.0013793103
    max: 28.5012068966
    count: 118
  }
  histogram_buckets {
    min: 28.5012068966
    max: 38.0010344828
    count: 67
  }
  histogram_buckets {
    min: 38.0010344828
    max: 47.500862069
    count: 15
  }
  histogram_buckets {
    min: 47.500862069
    max:[...]<jupyter_text>set target column<jupyter_code>! python dataset.py update_dataset --dataset_id TBL7687283924107526144 --target_column_spec_id 1490555137217789952<jupyter_output>Dataset updated. name: "projects/401832599639/locations/us-central1/datasets/TBL7687283924107526144"
display_name: "train"
create_time {
  seconds: 1564496871
  nanos: 340824000
}
etag: "AB3BwFq372u9hlP1Gpky6KuWjnHECjpwp98fF9qhhwEbNP4NWaCANjXvX-qx2QPnd_qj"
example_count: 27589
tables_dataset_metadata {
  primary_table_spec_id: "5409614800843964416"
  target_column_spec_id: "1490555137217789952"
  stats_update_time {
    seconds: 1564496993
    nanos: 135000000
  }
}<jupyter_text>model.py  list most recent model<jupyter_code>! python model.py list_models<jupyter_output>List of models:
Model name: projects/401832599639/locations/us-central1/models/TBL2266814744474157056
Model id: TBL2266814744474157056
Model display name: demo3_v2
Target column display name: price_per_mile
Training budget in node milli hours: 5000
Training cost in node milli hours: 1896
Model create time:
	seconds: 1563997966
	nanos: 318460000
Model deployment state: undeployed<jupyter_text>training model<jupyter_code>! python model.py create_model --dataset_id TBL7687283924107526144 --model_name demo3_v2 --train_budget_milli_node_hours 5000<jupyter_output>Training model...
Training operation name: projects/401832599639/locations/us-central1/operations/TBL3051905629124820992<jupyter_text>get status<jupyter_code>! python model.py get_operation_status --operation_full_id "projects/401832599639/locations/us-central1/operations/TBL3051905629124820992"<jupyter_output>Operation status: name: "projects/401832599639/locations/us-central1/operations/TBL3051905629124820992"
metadata {
  type_url: "type.googleapis.com/google.cloud.automl.v1beta1.OperationMetadata"
  value: "\032\014\010\220\253\201\352\005\020\210\346\222\352\002\"\014\010\255\357\201\352\005\020\220\341\313\313\003R\000"
}
done: true
response {
  type_url: "type.googleapis.com/google.cloud.automl.v1beta1.Model"
  value: "\nIprojects/401832599639/locations/us-central1/models/TBL1845165229361594368"
}<jupyter_text>list existing model again<jupyter_code>! python model.py list_models<jupyter_output>List of models:
Model name: projects/401832599639/locations/us-central1/models/TBL573461284582850560
Model id: TBL573461284582850560
Model display name: test_demo
Target column display name: price_per_mile
Training budget in node milli hours: 5000
Training cost in node milli hours: 2748
Model create time:
	seconds: 1564496817
	nanos: 269715000
Model deployment state: undeployed<jupyter_text>get model metadata<jupyter_code>! python model.py get_model --model_id TBL573461284582850560<jupyter_output>Model name: projects/401832599639/locations/us-central1/models/TBL573461284582850560
Model id: TBL573461284582850560
Model display name: test_demo
Model metadata:
target_column_spec {
  name: "projects/401832599639/locations/us-central1/datasets/TBL975513104441933824/tableSpecs/2068506827288477696/columnSpecs/6904655945503080448"
  data_type {
    type_code: FLOAT64
  }
  display_name: "price_per_mile"
}
input_feature_column_specs {
  name: "projects/401832599639/locations/us-central1/datasets/TBL975513104441933824/tableSpecs/2068506827288477696/columnSpecs/9210498954716774400"
  data_type {
    type_code: FLOAT64
  }
  display_name: "trip_miles"
}
input_feature_column_specs {
  name: "projects/401832599639/locations/us-central1/datasets/TBL975513104441933824/tableSpecs/2068506827288477696/columnSpecs/8057577450109927424"
  data_type {
    type_code: FLOAT64
  }
  display_name: "pickup_latitude"
}
input_feature_column_specs {
  name: "projects/401832599639/locations/us-central1/dataset[...]<jupyter_text>list existing model eval<jupyter_code>! python model.py list_model_evaluations --model_id TBL573461284582850560<jupyter_output>List of most recent model evaluations:
Model evaluation name: projects/401832599639/locations/us-central1/models/TBL573461284582850560/modelEvaluations/9119363279356331151
Model evaluation id: 9119363279356331151
Model evaluation example count: 2730
Model evaluation time:
	seconds: 1564506840
	nanos: 830440000<jupyter_text>get evaluation<jupyter_code>! python model.py get_model_evaluation --model_id TBL573461284582850560 --model_evaluation_id 9119363279356331151<jupyter_output>name: "projects/401832599639/locations/us-central1/models/TBL573461284582850560/modelEvaluations/9119363279356331151"
create_time {
  seconds: 1564506840
  nanos: 830440000
}
evaluated_example_count: 2730<jupyter_text>display evaluation<jupyter_code>! python model.py display_evaluation --model_id TBL573461284582850560<jupyter_output>Model regression metrics:
Model RMSE: 0.701361298561
Model MAE: 0.191221699119
Model MAPE: 2.81096339226
Model R^2: 0.96477496624<jupyter_text>deploy<jupyter_code>! python model.py deploy_model --model_id TBL573461284582850560<jupyter_output>Model deployed.<jupyter_text>predict.py  batch prediction<jupyter_code>! python predict.py batch_predict --model_id TBL573461284582850560 --input_path "bq://hackathon1-183523.demo3_v2.test" --output_path "bq://hackathon1-183523"<jupyter_output>Making batch prediction... 
Batch prediction complete.
create_time {
  seconds: 1564509019
  nanos: 115143000
}
update_time {
  seconds: 1564509172
  nanos: 622171000
}
batch_predict_details {
  input_config {
    bigquery_source {
      input_uri: "bq://hackathon1-183523.demo3_v2.test"
    }
  }
  output_info {
    bigquery_output_dataset: "bq://hackathon1-183523.prediction_test_demo_2019_07_30T10_50_19_032Z"
  }
}<jupyter_text>online prediction<jupyter_code>! python predict.py predict --model_id TBL573461284582850560 --file_path 'test.csv'
! curl -X POST -H "Content-Type: application/json" \
  -H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
  https://automl.googleapis.com/v1beta1/projects/hackathon1-183523/locations/us-central1/models/TBL6421666875700150272:predict \
  -d @request.json<jupyter_output>{
  "payload": [
    {
      "tables": {
        "value": 8.3914804458618164,
        "predictionInterval": {
          "start": 6.6885089874267578,
          "end": 12.913139343261719
        }
      }
    }
  ]
}
StarcoderdataJupyterStructuredCleanDedup