I've been having a really fun time building a site about pricing in my spare time. Here are a few example pages:




It's nowhere near finished, but I recently had the idea of playing around with price forecasting. Here are my thoughts: I will come up with several methods for forecasting price. I will treat each method of forecasting as an "experimental treatment", predict tomorrow's price with it, and over time, see which method is the most successful.

Right now I am using linear regression, and it's pretty terrible. ;)

Here are some questions that I would love feedback on for my experiment:

  1. It's not impossible to forecast price, right?
  2. Who out there is forecasting price successfully?
  3. Are there any papers, articles, etc. you could recommend on the subject of forecasting price?
  4. Anyone have a method they want me to try out? =]

I'd be happy to share my results over the next few months if anyone is interested.

  • This question has little to do with personal finance and is off-topic for money.SE. I vote to close it. – Dilip Sarwate Jul 31 '12 at 18:13

Assuming a price is set on an free market there are particular difficulties to pricing. A free market is one where the price is entirely determined by the willingness of people to buy and sell at a particular price point. What you perceive as price, is actually the "tick", i.e. the quote of the last transaction.

The first and most serious major obstacle to pricing is a variation of the prisoners dilemma, a psychological phenomenon. For instance, bitcoin might be worth 4$ now, but you believe it will be worth 5$ in 3 days. Will you buy bitcoin? If acting only on your conviction, yes. But what if you consider what other people will do? Will others believe bitcoin will be worth 5$ in 3 days? Will they act on their conviction? Will the others believe that others believe that it wil be worth 5$ in 3 days, and will the others believe that the others who believe will act on their conviction? Will the others believe that others believe of still others who believe that they will act on their conviction? It goes on like this ad-infinitum.

The actual behavior of any individual on the market is essentially chaotic and unpredictable (for the reason stated above and others).

This is related to a phenomenon you call market efficiency. An efficient market always reflects the optimal price-point at any given time. If that is so, then you cannot win on this market, because at the time you would have to realize a competitive edge, everybody else has already acted on that information.

Markets are not 100% efficient of course. But modern electronic markets can be very, very efficient (as say compared to stock markets fro 100 years ago, where you could get a competitive edge just by having access to a fast courier).

What makes matters rather more difficult for price forecasting is that not only are humans engaging in the market, machines are as well. The machines may not be terribly good at what they do, but they are terribly fast. The machines that work well (i.e. don't loose much) will survive, and the ones that don't will die in short order. Since speed is one of the major benefits of the machines over humans, they tend to make markets even more efficient.

Another phenomenon to price forecasting is that of information and entropy. Suppose you found a reliable method to predict a market at a given time. You act on this information and indeed you make a profit. The profit you will be able to achieve will diminish over time until it reaches zero or reverts. The reason for this is that you acted on private information, which you leaked out by engaging in a trade. The more successful you are in exploiting your forecast, the better you train every other market participant to react to their losses. Since for every trade you make successfully, there has to be somebody who lost. People or machines who lose on markets usually exit those markets in some fashion. So even if the other participants are not adjusting their behavior, your success is weeding out those with the wrong behavior.

Yet another difficulty in pricing forecasts are black-swan events. Since information can have a huge impact on pricing, the sudden appearance of new information can throw a conservative forecast completely off the rails and incur huge losses (or huge unexpected benefits). You cannot quantify black-swan events in any shape or form.

It is my belief that you cannot predict efficient and well working markets. You might be able to predict some very sub-optimal markets, but usually, hedge-funds are always on the hunt for inefficient markets to exploit, so by simple decree of market economics, the inefficient markets tend to be a perpetually dying species.


well there are many papers on power spot price prediction, for example. It depends on what level of methodology you would like to use. Linear regression is one of the basic steps, then you can continue with more advanced options. I'm a phd student studying modelling the energy price (electricity, gas, oil) as stochastic process.

Regarding to your questions: 1. mildly speaking, it's really hard, due to its random nature! (http://www.dataversity.net/is-there-such-a-thing-as-predictive-analytics/) 2. well, i would ask what kind of measure of success you mean? what level of predicted interval one could find successful enough? 3. would you like me to send you some of the math-based papers on? 4. as i know, the method is to fully capture all main characteristics of the price. If it's daily power price, then these are mean-reversion effect, high volatility, spike, seasonality (weekly, monthly, yearly).

Would you tell me what kind of method you're using? Maybe we can discuss some shared ideas?



It's not impossible to forecast the future price of a commodity. However, it's exactly that; an educated guess, much like the weather, and the further out that prediction is made, the higher the percentage error is expected. A lot of information is gathered by various instruments, spotters etc at a very high cost of time and money, to produce a prediction that starts breaking down after about five days and is no more than a wild guess after about ten.

How accurately a price can be forecast depends on the commodity. There are seasonal and thus cyclical changes in many commodities, on top of which there is a general trend which is nearer term. A pretty decent prediction can thus be arrived at with a relatively simple seasonally-adjusted percentage change algorithm; take a moving average of the last few measurements, compute the percent change versus the same period last year (current minus last divided by last) and multiply it by last year's number for the current day or month to arrive at a pretty decent prediction for the current and near-future periods (up to about as far ahead as you have looked behind).

Another thing you may need to do is normalize. Many price graphs are very jittery; the price of a stock may fluctuate many percentage points on a single day, and there's a lot of "noise" inherent in them. A common tool to normalize is a box-and-whisker plot, which for a given time period will aggregate all samples within that period, and give you a measurement of the lowest sample, highest sample, median, and quartiles (the range of each 25% of the full sample space). Box plots can also be plotted on the "interquartile range" or "middle fifty"; this throws away the very noisy outliers and constructs a much more regular plot from the inner part of the bell curve. You can reverse-engineer a best-fit line connecting the elements of each box, and the closer two lines are, the more likely the real future data will be around that area (because the quartile between those to lines is very dense; 25% of the values are in a very small range meaning many samples occurred there).

Lastly, there are outside factors that are not included in simple percentage growth. Big news must be taken into account by introducing more subjective guesses about future data. If you see an active hurricane season coming (or a hurricane bearing down on Galveston/Houston) then it's reasonable to assume that the price of oil and/or refined oil products (like gas and jet fuel) will skyrocket. A cyclical growth model will not predict these events, but you can factor in the likelihood of a big change with a base onto which you add last year's numbers, and onto that you add regular growth. Conversely, when a huge spike happens due to a non-cyclical event like a natural disaster, you must smooth it out by reducing the readings to fit in the curve, otherwise your model for next year will expect the same anomaly at the same time and so it will be wrong. These adjustments are necessary, but the more of them you make, the less the graph reflects real history and the more it reflects what you think it should have been.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.