John Regehr writes about squeezing a research idea, an exercise for evaluating a research idea. A squeeze is like a feasibility study. An idea can also be squeezed after the fact to determine its value or identify future work for improvements. The basic idea is to find a bound on the performance improvement of the research idea.
My experience has been that the second step is best accomplished by developing a model that stresses the enhancement proposed in the research idea. Exploring system behavior in the limit puts the proposed enhancement in perspective. Upper bounds on the usefulness of the idea can be determined by idealizing constraint parameters of the model. See applications of Amdahl's law for some examples of doing this right.
A metric that is easy to measure, explain, and visualize should be applied to the baseline and enhanced system. A good metric for evaluating a squeeze will also be useful in presenting results and selling the research project. The measures taken of the two systems provide the last step of the squeeze.
In systems work, this analysis is often seen when results are normalized to a baseline, such as a benchmark on a stock system. Typical metrics in final results include speedup, throughput (bandwidth), and latency. Casting a squeeze in similar language as final results eases transitioning from pilot studies to experiments to disseminating results.
Squeezing an idea is a way to determine if going forward with it makes sense. The results of the squeeze will be useful in motivating the contribution of the research idea. Gaps between experimental results and predicted upper bounds will provide areas for further improvement.
To squeeze an idea you ask:A good baseline, or control in more classical terms, yields the first step of a squeeze. Baseline results should represent state-of-the-art solutions that are readily available. Comparing a proposed idea to an in-house or contrived baseline often does not make for good science.
- How much of the benefit can be attained without the new idea?
- If the new idea succeeds wildly, how much benefit can be attained?
- How large is the gap between these two?
My experience has been that the second step is best accomplished by developing a model that stresses the enhancement proposed in the research idea. Exploring system behavior in the limit puts the proposed enhancement in perspective. Upper bounds on the usefulness of the idea can be determined by idealizing constraint parameters of the model. See applications of Amdahl's law for some examples of doing this right.
A metric that is easy to measure, explain, and visualize should be applied to the baseline and enhanced system. A good metric for evaluating a squeeze will also be useful in presenting results and selling the research project. The measures taken of the two systems provide the last step of the squeeze.
In systems work, this analysis is often seen when results are normalized to a baseline, such as a benchmark on a stock system. Typical metrics in final results include speedup, throughput (bandwidth), and latency. Casting a squeeze in similar language as final results eases transitioning from pilot studies to experiments to disseminating results.
Squeezing an idea is a way to determine if going forward with it makes sense. The results of the squeeze will be useful in motivating the contribution of the research idea. Gaps between experimental results and predicted upper bounds will provide areas for further improvement.
0 comments:
Post a Comment