Jump to content

Welcome to the new Traders Laboratory! Please bear with us as we finish the migration over the next few days. If you find any issues, want to leave feedback, get in touch with us, or offer suggestions please post to the Support forum here.

  • Welcome Guests

    Welcome. You are currently viewing the forum as a guest which does not give you access to all the great features at Traders Laboratory such as interacting with members, access to all forums, downloading attachments, and eligibility to win free giveaways. Registration is fast, simple and absolutely free. Create a FREE Traders Laboratory account here.

SNYP40A1

Best Open-source Data Smoother?

Recommended Posts

I am looking for an open-source smoother to down-sample data. I know there are many such as Hull Moving average, but I have data logged at the ms scale at non-fixed time intervals and I want to sample it on the second / minute / hour etc. range at fixed time intervals. Does anyone know of a good smoother to do this? The key is that the input data is not logged at a constant frequency.

Share this post


Link to post
Share on other sites

Aren't sampling and smoothing different issues? Not sure exactly what you want to end up with. The 'normal' way of dealing with the data would be to sample it (highest high lowest low of sample period) then smooth it. You could of course smooth the data in each sample first (e.g. take an hours data average it and use that price as your sample value for that discrete period)

 

As to actual smoothing algorithms there are whole bunch here (and other places too).

Share this post


Link to post
Share on other sites

It's basically an exponential moving average...that's what would allow for smoothing and downsampling in the same pass. The idea is that as data-points start coming in, they are entered into a distribution such that each datapoint's weight in the distribution decreases with time. Then the average or preferably median of the distribution can be sampled at any time to get the value most closely representing the real-time sample average. One way to get the sample is to re-evaluate all data in the distribution to find whatever info is of interest. But I think there is a more efficient way, that's what I am looking for. Key is smoothing function where the input data does not have a fixed period. Used for plotting tick data on time vs. price chart where each pixel may correspond to seconds or minutes rather than microseconds. How do most software packages handle this issue? I think they just group everything into time periods and then average the time period, as you suggested. That would provide some smoothing...it just seems like it would be more accurate to smooth and down-sample at the same time rather than smoothing on already-downsampled data. I'll think about this some more...the accuracy difference might be insignificant.

Share this post


Link to post
Share on other sites

Are you weighting by time or actually by position in the time series? Will later data effect weights of earlier data (that becomes processor intensive). Just a couple of things to consider.

 

What platform btw?

Share this post


Link to post
Share on other sites

The keyword to search for in the literature is "non-uniform sampling," and most algorithms I'm aware of interpolate to find uniform samples and then do typical DSP (choose your favorite low-pass filter) on the uniform samples. How accurate you need the results to be will dictate how complex the interpolation process is. Of course for some purposes you could apply a (possibly non-linear) least-squares fitted model, or a spline etc. to the data, and call the resulting model the "smoothed" data with no further filtering.

Edited by RichardTodd
added model suggestion

Share this post


Link to post
Share on other sites
Are you weighting by time or actually by position in the time series? Will later data effect weights of earlier data (that becomes processor intensive). Just a couple of things to consider.

 

What platform btw?

 

Weighing by time rather than position in the time series would be more accurate. If I weighted only by position, then I could basically apply any moving average -- that would essentially be treating the data as uniform sampled. I could go even more complicated. I could weight the data by time as well as volume to reflect more of the instantaneous "force" of the market. Or weight by bid vs. ask bias, etc. Although, at that point, I have a new indicator beyond just plotting the data. For now I simply want to feed in the tick data and for any given point in time, get an estimate for what the price is at that point. What I plan to do now is simply take the last trade behind my desired sample point. Been a while since I took a signals and systems class, but I think this process is referred to as zero-order hold:

 

Continuous/Discrete Conversions of LTI Models :: Operations on LTI Models (Control System Toolbox™)

 

I am developing my own platform in Java -- at least the analysis part.

Share this post


Link to post
Share on other sites
The keyword to search for in the literature is "non-uniform sampling," and most algorithms I'm aware of interpolate to find uniform samples and then do typical DSP (choose your favorite low-pass filter) on the uniform samples. How accurate you need the results to be will dictate how complex the interpolation process is. Of course for some purposes you could apply a (possibly non-linear) least-squares fitted model, or a spline etc. to the data, and call the resulting model the "smoothed" data with no further filtering.

 

Non-uniform sampling. That's what I am looking for. I thought about doing least-squares analysis / linear regression, but I think that's probably getting too complicated...I guess all I am really trying to do at this point is plot the data. In Ninjatrader, it looks like what they do is simply group all the data into time intervals (1-minute, 5-minute, 233 tick, etc) and then plot the close of the given bar and hold that level until the next close. Essentially, they are doing zero-order hold. That's simple and it works, but seems like there must be an algorithm I could use and get a much smoother plot for very little effort.

Share this post


Link to post
Share on other sites

SSA (singular spectrum analysis) is a tool that is often used to fill in the gaps on univariate time series data. Then you can sample in fixed time intervals. The problem off course is that this technique is not causal. So you can't use it in real time, but it will work fine on historic data.

Share this post


Link to post
Share on other sites

Actually, the longer I look at this problem, I don't see anything wrong with applying a moving average to the data and ignoring the non-uniform sampling issue. Run a moving average and at the given point of interest, and just take the last update of the MA. The reason that I think this approach is valid is because signal reconstruction, as long as the Nyquist Criterion applies, does not require uniform sampling -- I read that statement somewhere on comp.dsp, but I have not been able to verify it. Can anyone confirm?

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


  • Topics

  • Posts

    • PLTR Palantir Technologies stock watch, consolidation at the 81.29 triple support area at https://stockconsultant.com/?PLTR
    • WMT Walmart stock watch, big pullback to 84.84 support area with high trade quality at https://stockconsultant.com/?WMT
    • AXSM Axsome Therapeutics stock watch. pullback to 120.5 gap support area with bullish indicators at https://stockconsultant.com/?AXSM
    • Date: 14th March 2025.   Gold Surges Past Record High as Market Volatility Persists.     Gold Surges Past Record High as Market Volatility Persists Gold surged towards $2,994 per ounce, surpassing its previous high set on Thursday. With a 2.6% rise this week, gold is on track for its most significant gain since November. Meanwhile, gold futures in New York comfortably exceeded the $3,000 mark, reflecting strong investor sentiment toward the precious metal. The robust performance of gold this quarter extends its strong annual rally in 2024. Market uncertainty, exacerbated by the US administration’s aggressive trade policies, has dampened risk appetite for equities, pushing the S&P 500 into correction territory this week. Central bank purchases increased ETF inflows, and bullish forecasts from major banks have further fueled gold’s ascent. Trade Tensions and Market Impact Former President Donald Trump escalated trade tensions by proposing a 200% tariff on European alcoholic beverages, including wine and champagne. Additionally, he reaffirmed his stance on retaining tariffs on steel and aluminium and signalled that reciprocal tariffs on global trade partners could take effect as early as April 2. As we approach the second quarter, reciprocal tariffs could drive another wave of market turbulence, solidifying gold’s appeal as a safe-haven asset. Gold and Equity Market Reactions The upward momentum in gold has also lifted mining stocks, with Australia’s Evolution Mining Ltd. reaching an all-time high. Global holdings in gold-backed ETFs increased to 2,687 tons, marking the highest level since November 2023. Analysts at major banks remain bullish on gold’s trajectory. Macquarie Group recently forecasted a potential spike to $3,500 per ounce in Q2, while BNP Paribas revised its outlook to show gold prices consistently above $3,000. Gold traded at $2,983.50 per ounce in the Asia session, reflecting a 14% year-to-date gain. Meanwhile, silver edged lower after nearing $34 per ounce, while platinum and palladium recorded gains.     US Stock Market Recovery Amid Uncertainty After a sharp sell-off, US stock futures rebounded. Futures tied to the Dow Jones Industrial Average rose 0.4%, while S&P 500 and Nasdaq Composite futures gained 0.6% and 0.8%, respectively. Despite the slight recovery, Wall Street remains on edge following the S&P 500’s descent into correction territory. Trump’s firm stance on tariffs has added to market concerns. During a meeting with NATO’s secretary general, he dismissed any possibility of easing trade restrictions, acknowledging that further market disruptions may lie ahead. Government Shutdown and Economic Indicators Adding to the economic uncertainty, a potential US government shutdown loomed over Wall Street. However, a breakthrough emerged late Thursday as Senate Minority Leader Chuck Schumer signalled a willingness to advance a Republican-led stopgap spending bill. Today the University of Michigan’s consumer sentiment survey is expected to shed light on how consumers are coping with inflation and trade disruptions. Last month’s report indicated weakening economic confidence, which could have further implications for spending trends. Asian Markets Rally Amid China’s Economic Stimulus Asian stock markets saw a strong performance this morning, brushing off Wall Street’s losses. Chinese stocks surged after state-run banks and financial institutions were instructed to support consumer spending. Hong Kong’s Hang Seng Index jumped 2.5% to 24,038.85, while the Shanghai Composite Index gained 1.9% to 3,420.65. In Tokyo, the Nikkei 225 added 0.9%, while Australia’s ASX 200 climbed 0.6%. China’s National Financial Regulatory Administration issued directives aimed at boosting consumer finance, including encouraging credit card usage and providing support for struggling borrowers. Economists, however, argue that broader reforms—such as wage growth and enhanced social welfare—are necessary for sustained economic recovery. Wall Street’s Struggles Amid AI Stock Declines Despite positive economic data, including lower-than-expected wholesale inflation and strong job market indicators, stock market turbulence continued. AI-related stocks, which have been at the forefront of market gains, faced renewed pressure. Palantir Technologies fell 4.8%, Super Micro Computer dropped 8%, and Nvidia fluctuated before closing 0.1% lower. Tesla also struggled, declining 3% and extending its 2025 losses to over 40%. In contrast, Intel shares soared 14.6% after announcing Lip-Bu Tan as its new CEO. Oil Prices and Currency Movements In commodities, US crude oil prices rose by $0.46 to $67.01 per barrel, while Brent crude increased by $0.44 to $70.32 per barrel. The US dollar strengthened to 148.63 Yen, while the Euro dipped slightly to $1.0845. Conclusion Market volatility remains high as investors navigate shifting trade policies, inflation concerns, and economic uncertainties. While gold continues to shine as a safe-haven asset, equity markets face persistent headwinds. As geopolitical and economic developments unfold, traders and investors must remain vigilant in the days ahead.   Always trade with strict risk management. Your capital is the single most important aspect of your trading business.   Please note that times displayed based on local time zone and are from time of writing this report.   Click HERE to access the full HFM Economic calendar.   Want to learn to trade and analyse the markets? Join our webinars and get analysis and trading ideas combined with better understanding of how markets work. Click HERE to register for FREE!   Click HERE to READ more Market news.   Andria Pichidi HFMarkets   Disclaimer: This material is provided as a general marketing communication for information purposes only and does not constitute an independent investment research. Nothing in this communication contains, or should be considered as containing, an investment advice or an investment recommendation or a solicitation for the purpose of buying or selling of any financial instrument. All information provided is gathered from reputable sources and any information containing an indication of past performance is not a guarantee or reliable indicator of future performance. Users acknowledge that any investment in Leveraged Products is characterized by a certain degree of uncertainty and that any investment of this nature involves a high level of risk for which the users are solely responsible and liable. We assume no liability for any loss arising from any investment made based on the information provided in this communication. This communication must not be reproduced or further distributed without our prior written permission.
    • ItuGlobal: Our Latest NETELLER VIPs 2025 ITU GLOBAL VIP Members’ Rewards Every calendar year, we choose 2 customers to become our VIPs. They’ve permanently special status with us and they can fund/withdraw Neteller through us, at parallel market rates, whether they open brokerage accounts through us or not.   These are people who funded with the highest amount of Neteller, and who also withdrew the highest amount of Neteller through us.  They would be announced in January each year and added to our list of VIPs. ItuGlobal: Our Latest NETELLER VIPs 2025 Adetoye Oyebanji Babalola: Adetoye O. has started selling large quantities of Neteller to us since the very beginning of our company’s existence. He also sold Perfect Money to us many times, when PM was still available in Nigeria. Besides, he has given us a lot of helpful business advice, which has proven to be invaluable to us. He deserves to become our VIP. Isiaka Adekunle Mohammed: He is a constant buyer. Buying e-currencies and also funding his Instaforex account through us. We thank Isiaka A. for his trust in us and wish him the best in everything he does. Abiodun Lawanson: This is an avid buyer and seller of Neteller. He buys and makes profits and sells back to us. Sometimes when we are not online, he will send an offline message and we will process his order once we come back online. He has thus become our VIP. Source: Ituglobalfx.com.ng  
×
×
  • Create New...

Important Information

By using this site, you agree to our Terms of Use.