Skip to content

Commit 44c5b63

Browse files
jvpcmsjvpcms
committed
qualitative investigation perf_ratio values
1 parent 15ea84f commit 44c5b63

1 file changed

Lines changed: 136 additions & 31 deletions

File tree

‎examples/Simulated_Annealing/simulated_annealing.ipynb‎

Lines changed: 136 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -630,18 +630,6 @@
630630
" pickle.dump(bs_df, open(file_path, \"wb\"))"
631631
]
632632
},
633-
{
634-
"cell_type": "markdown",
635-
"id": "d74b6ac1",
636-
"metadata": {},
637-
"source": [
638-
"### Resource\n",
639-
"\n",
640-
"In this analysis, we examine how the approximation ratio evolves as the number of sweeps increases during algorithm execution.\n",
641-
"\n",
642-
"The number of sweeps serves as our resource metric. The objective is to achieve a high approximation ratio while minimizing the number of sweeps required (i.e., using fewer computational resources)."
643-
]
644-
},
645633
{
646634
"cell_type": "code",
647635
"execution_count": 75,
@@ -678,28 +666,15 @@
678666
]
679667
},
680668
{
681-
"cell_type": "code",
682-
"execution_count": 76,
683-
"id": "c88612ee",
669+
"cell_type": "markdown",
670+
"id": "d74b6ac1",
684671
"metadata": {},
685-
"outputs": [],
686672
"source": [
687-
"# Load Bootstrap data.\n",
688-
"# The data is already boostrapped, but needs to be loaded into memory\n",
673+
"### Resource\n",
689674
"\n",
690-
"shared_args = {\n",
691-
" \"response_col\": \"PerfRatio\",\n",
692-
" \"resource_col\": \"resource\",\n",
693-
" \"response_dir\": 1,\n",
694-
" \"confidence_level\": 68,\n",
695-
"}\n",
696-
"boots_range = [1, 10, 100, 1000]\n",
697-
"bsParams = bootstrap.BootstrapParameters(\n",
698-
" shared_args=shared_args, update_rule=lambda df: None\n",
699-
")\n",
700-
"bs_iter_class = bootstrap.BSParams_range_iter()\n",
701-
"bsParams_iter = bs_iter_class(bsParams, boots_range)\n",
702-
"sb.run_Bootstrap(bsParams_iter)"
675+
"In this analysis, we examine how the approximation ratio evolves as the number of sweeps and reads increases during algorithm execution.\n",
676+
"\n",
677+
"The number of sweeps times the number of reads serves as our resource metric. This corresponds to the total time spent in a SA run. The objective is to achieve a high approximation ratio while minimizing that product (i.e., using fewer computational resources)."
703678
]
704679
},
705680
{
@@ -745,6 +720,136 @@
745720
"sb.run_Interpolate(iParams)"
746721
]
747722
},
723+
{
724+
"cell_type": "markdown",
725+
"id": "7b306e59",
726+
"metadata": {},
727+
"source": [
728+
"### Initial Investigation\n",
729+
"\n",
730+
"For instance 0 of the problem, let's investigate where the performance ratio reaches the extreme values, 0 and 1, across the various numbers of sweeps."
731+
]
732+
},
733+
{
734+
"cell_type": "code",
735+
"execution_count": 109,
736+
"id": "cf98939a",
737+
"metadata": {},
738+
"outputs": [],
739+
"source": [
740+
"def filter_df_by_reads(df, reads):\n",
741+
" return df[df[\"reads\"] == reads]"
742+
]
743+
},
744+
{
745+
"cell_type": "code",
746+
"execution_count": 110,
747+
"id": "774ce811",
748+
"metadata": {},
749+
"outputs": [],
750+
"source": [
751+
"def find_sweep_with_min_perf_ratio(df) -> tuple[int, float]:\n",
752+
" min_perf_ratio = df[\"Key=PerfRatio\"].min()\n",
753+
" return df[df[\"Key=PerfRatio\"] == min_perf_ratio][\"sweeps\"].values[0], min_perf_ratio\n",
754+
"\n",
755+
"def find_sweep_with_max_perf_ratio(df) -> tuple[int, float]:\n",
756+
" max_perf_ratio = df[\"Key=PerfRatio\"].max()\n",
757+
" return df[df[\"Key=PerfRatio\"] == max_perf_ratio][\"sweeps\"].values[0], max_perf_ratio"
758+
]
759+
},
760+
{
761+
"cell_type": "code",
762+
"execution_count": 111,
763+
"id": "c88612ee",
764+
"metadata": {},
765+
"outputs": [],
766+
"source": [
767+
"# Load Bootstrap data.\n",
768+
"# The data is already boostrapped, but needs to be loaded into memory\n",
769+
"\n",
770+
"shared_args = {\n",
771+
" \"response_col\": \"PerfRatio\",\n",
772+
" \"resource_col\": \"resource\",\n",
773+
" \"response_dir\": 1,\n",
774+
" \"confidence_level\": 68,\n",
775+
"}\n",
776+
"boots_range = [1, 10, 100, 1000]\n",
777+
"bsParams = bootstrap.BootstrapParameters(\n",
778+
" shared_args=shared_args, update_rule=lambda df: None\n",
779+
")\n",
780+
"bs_iter_class = bootstrap.BSParams_range_iter()\n",
781+
"bsParams_iter = bs_iter_class(bsParams, boots_range)\n",
782+
"sb.run_Bootstrap(bsParams_iter)"
783+
]
784+
},
785+
{
786+
"cell_type": "code",
787+
"execution_count": 112,
788+
"id": "ea293d80",
789+
"metadata": {},
790+
"outputs": [
791+
{
792+
"name": "stdout",
793+
"output_type": "stream",
794+
"text": [
795+
"Reads |Min Perf Ratio |At |Max Perf Ratio |At \n",
796+
"--------------------|--------------------|--------------------|--------------------|--------------------\n",
797+
"1000 |0.8399084864502725 |1.0 |1.0 |22.0 \n",
798+
"--------------------|--------------------|--------------------|--------------------|--------------------\n",
799+
"100 |0.8060603148994175 |1.0 |1.0 |107.0 \n",
800+
"--------------------|--------------------|--------------------|--------------------|--------------------\n",
801+
"10 |0.7499112055464742 |1.0 |0.9999558217089615 |820.0 \n",
802+
"--------------------|--------------------|--------------------|--------------------|--------------------\n",
803+
"1 |0.6617116690688953 |1.0 |0.9795630428412597 |820.0 \n",
804+
"--------------------|--------------------|--------------------|--------------------|--------------------\n"
805+
]
806+
}
807+
],
808+
"source": [
809+
"bootstrapped_data_path = os.path.join(checkpoints_path, f'bootstrapped_results_inst=0.pkl')\n",
810+
"bootstrapped_data = pickle.load(open(bootstrapped_data_path, 'rb'))\n",
811+
"\n",
812+
"print(f\"{'Reads':<20}|{'Min Perf Ratio':<20}|{'At':<20}|{'Max Perf Ratio':<20}|{'At':<20}\")\n",
813+
"print(\"-\"*20 + \"|\" + \"-\"*20 + \"|\" + \"-\"*20 + \"|\" + \"-\"*20 + \"|\" + \"-\"*20)\n",
814+
"\n",
815+
"df_1000 = filter_df_by_reads(bootstrapped_data, 1000)\n",
816+
"min_perf_ratio_sweep, min_perf_ratio = find_sweep_with_min_perf_ratio(df_1000)\n",
817+
"max_perf_ratio_sweep, max_perf_ratio = find_sweep_with_max_perf_ratio(df_1000)\n",
818+
"print(f\"{1000:<20}|{min_perf_ratio:<20}|{min_perf_ratio_sweep:<20}|{max_perf_ratio:<20}|{max_perf_ratio_sweep:<20}\")\n",
819+
"print(\"-\"*20 + \"|\" + \"-\"*20 + \"|\" + \"-\"*20 + \"|\" + \"-\"*20 + \"|\" + \"-\"*20)\n",
820+
"\n",
821+
"df_100 = filter_df_by_reads(bootstrapped_data, 100)\n",
822+
"min_perf_ratio_sweep, min_perf_ratio = find_sweep_with_min_perf_ratio(df_100)\n",
823+
"max_perf_ratio_sweep, max_perf_ratio = find_sweep_with_max_perf_ratio(df_100)\n",
824+
"print(f\"{100:<20}|{min_perf_ratio:<20}|{min_perf_ratio_sweep:<20}|{max_perf_ratio:<20}|{max_perf_ratio_sweep:<20}\")\n",
825+
"print(\"-\"*20 + \"|\" + \"-\"*20 + \"|\" + \"-\"*20 + \"|\" + \"-\"*20 + \"|\" + \"-\"*20)\n",
826+
"\n",
827+
"df_10 = filter_df_by_reads(bootstrapped_data, 10)\n",
828+
"min_perf_ratio_sweep, min_perf_ratio = find_sweep_with_min_perf_ratio(df_10)\n",
829+
"max_perf_ratio_sweep, max_perf_ratio = find_sweep_with_max_perf_ratio(df_10)\n",
830+
"print(f\"{10:<20}|{min_perf_ratio:<20}|{min_perf_ratio_sweep:<20}|{max_perf_ratio:<20}|{max_perf_ratio_sweep:<20}\")\n",
831+
"print(\"-\"*20 + \"|\" + \"-\"*20 + \"|\" + \"-\"*20 + \"|\" + \"-\"*20 + \"|\" + \"-\"*20)\n",
832+
"\n",
833+
"df_1 = filter_df_by_reads(bootstrapped_data, 1)\n",
834+
"min_perf_ratio_sweep, min_perf_ratio = find_sweep_with_min_perf_ratio(df_1)\n",
835+
"max_perf_ratio_sweep, max_perf_ratio = find_sweep_with_max_perf_ratio(df_1)\n",
836+
"print(f\"{1:<20}|{min_perf_ratio:<20}|{min_perf_ratio_sweep:<20}|{max_perf_ratio:<20}|{max_perf_ratio_sweep:<20}\")\n",
837+
"print(\"-\"*20 + \"|\" + \"-\"*20 + \"|\" + \"-\"*20 + \"|\" + \"-\"*20 + \"|\" + \"-\"*20)"
838+
]
839+
},
840+
{
841+
"cell_type": "markdown",
842+
"id": "5363d6b7",
843+
"metadata": {},
844+
"source": [
845+
"This first analysis indicates that choosing a large number of reads is a good strategy given a certain amount of finite resource. However, this still doesn't paint the entire picture.\n",
846+
"\n",
847+
"- How to \"distribute\" the parameters given the resources?\n",
848+
"- How close to the virtual best does a certain amount of available resources get us?\n",
849+
"\n",
850+
"These questions will be answered by the complete framework evaluation."
851+
]
852+
},
748853
{
749854
"cell_type": "code",
750855
"execution_count": 78,

0 commit comments

Comments
 (0)