Carlos, since I reverted to the Python 2 version (which is working perfectly) I can't be certain which version of PS3 I was using. I believe I saved the PS3 version I was using in a new folder. Based on that, here is the last of the change log in that setup.py file: #14Dec2023*cgn # - Update to python3
# Define the required minimum Python version. Make check compatible with Python 2 since # a lot of users are coming from 2.7.15 import sys REQUIRED_MAJOR = 3 REQUIRED_MINOR = 10
I had emailed Cal directly with this issue. I don't know if his response would be useful for you: "I don't have a theory on why that behavior. Hard to imagine what Python code would cause that truncation. I would tend to suspect an overly greedy RegEx."
Ken, as luck would have it, I just recently started using Discover again and I noticed new transactions were not coming in. Behold, the same issue you experienced. FITID getting truncated! Doesn't happen with any other credit cards. Weird. Hopefully how that I have a test case I can track down what is going on! Thank you again for pointing it out.
At least in my case, I never stopped running the Discover Card scrubber that came with the original PocketSense 2. That scrubber dates back to direct OFX downloads and Discover would do some wonky stuff with their FITIDs which Pocket Sense 2 attempted to fix. Preliminary conclusion is that that scrubber is messing up the outputs of the BoA MFP ofx files, which do not have FITID issues. I'll have to go track down why the behavior is different in PocketSense3 and PocketSense2, but I think easiest solution will be to eliminate the scrubber altogether, as it serves no other purpose.
I wasn't subscribed to this continuation so I missed your replies. Guess I should have checked before I did a little detective work and came to the same conclusion. Cal had asked me to negate the scrubbers as a possible cause and after finding out getdata doesn't work if scrubber.py isn't active, I determined that scrub_CL7.py isn't a problem and scrub_discover.py causes truncation. Since I'm not a Python programmer I'm going to leave it to you and Cal to figure out why scrub_discover.py works with PS2 and truncates with PS3. I'm back using your PS3 version without any issue (that I've found so far). By the way, I use Jon Larssen's BoAMFP to generate OFX files for all non-Fidelity bank and credit card accounts (including Discover; which is now Capital One!)
I had earlier posted an issue I was finding of the final price of the day not updating the middle of the day transactions for ETFs if there was a transaction on the last day of the downloaded statement. This seems to be because of Money getting confused with time stamps.
The transaction and the portfolio position in the generated OFX have the same time stamp. The downloaded CSV files from Fidelity don't seem to have a timestamp, only a date. So, I believe F2O generates a timestamp for DTAS something like 20260821120000. (If this is UTC, then it seems incorrect for US time closing but that isn't the issue). The time is the same for all entries both for the transaction on the last day and the portfolio position. This is why Money is often using the transaction price as the final price. For Mutual Funds this does not matter since the last day price and last day transaction are both at the end of the day and the same. But for ETFs traded during the day this makes a difference. Don't know what the solution to this would be.
GV: With F2O, it is best to do the downloads before the market closes, or to wait until the mutual funds have been priced... which might be as late as 9PM. Before market close (including before market open) computes the close for the previous day, and uses that with the appropriate date. I tend to use the next morning or on the weekend.
So while the market is open (until 4pm ET), the previous close is used.
The deal with dates like 20260821120000 is the 120000 represents a time of day of 12 noon. That will present the correct date regardless of time zone. With other times, Money can compute a wrong date.
Here is a quote from the OFX 1.6 OFX specification: :Note: Developers should consider the possibility of a date change due to timezone conversion. A datetime value in the GMT timezone with a time of 12:00:00 (noon) would be converted to another time on the same date in every timezone. For example, 199812251200 remains Christmas Day in every timezone."
GV: It occurs to me that you might want F2O to bring in intraday prices. If prices come in via OFX, Money will not accept a later update for price for that day. That is the way it is, and it is the reason that F2O attempts to only bring in closing prices with the appropriate date for the close.
I would note that when Fidelity provided direct OFX, the dates could be a bit off... like having prices dated on Saturday, which never happens for real.
If you want intraday prices, I suggest MSMoneyQuotes. https://gaiersoftware.com/Money The price is a one-time $10. Very reasonable. Free trial. MSMoneyQuotes makes use of the internal quote feature that Money had had when it fetched prices on its own, rather than getting quotes via OFX. These quotes can be overwritten by later downloads.
The context of my previous post was not clear. I am NOT doing any intraday downloads. I only download on a Sunday to bring my MSM up to date and all prices will have stabilized by then. Let us keep this as the context for the issue. Forget mutual funds. Consider only ETFs.
Suppose there is an ETF transaction intraday on the Fri before the Sun download. This transaction is done at the intraday price where it occurred. Assume that this is somewhere midday, so the final price of the ETF at the end of the day is different from the transaction price. To record this transaction at the price of the transaction, the activity log entry is used. This as downloaded from Fidelity on the Sunday has a date but not a time stamp. Which means, the timestamp generated for the transaction in the OFX file has the date for that Fri and a generated timestamp of 120000 by F2O (as far as I can see). When Money sees this transaction in the import it creates that transaction and updates the price list with that price of the transaction (intraday) FOR THAT FRI DATE as a buy or sell depending on the transaction. The portfolio download on the other hand has the final price for the Fri. But since the download only has the date and not time, the same timestamp as the transaction is generated for the portfolio so both the portfolio item in OFX and the transaction item in OFX have the exact same date and time but with two different prices.
If MSM sees the transaction item first and records that midday price for the Fri, when it encounters the portfoilio position item with the same timestamp and the final price at the end of the day Fri, it seems just to drop the latter since a price for that timestamp has already been encountered. It will not update the price twice for the same day. So you land up with that ETF as having the transaction price as the last price recorded on Fri rather than its final price for the position. So the total portfolio value will not match the total in Fidelity on Sun. It as to be manually updated to the last price of the Fri.
If you generate a time stamp of 120000 for transaction and 120001 for portfolio position item, then I am thinking that Money will see the portfolio position as being at a later timestamp than transaction and update the price again. So, you will have the correct portfolio totals on Sun to start the next week.
My experiments with existing files show that if there is a Buy or Sell for a date, Money ignores the downloaded price for that date.
You can find the price history by going to the investment details for an investment. UpdatePrices->UpdatePricesManually Look through the table. You will see that if there is a Buy or a Sell, there is no Update or Oneline price listed for that date. The price shown is for the latest-entered Buy or Sell for that day.
In OFX, Money uses the first <DTASOF> as the date of quotes.
While I doubt making the change will matter, I may try that at some point. If you want to try it out, you can just change one line.
Line 1291 of the latest F2O.py contains dtasof_ele=("<DTASOF>"+dt_Ymd+"120000\n") #coulf move \n
You could change that line, using Notepad, to be this:
dtasof_ele=("<DTASOF>"+dt_Ymd+"120001\n") #modified as test
Note that time 120000 is noon GMT/UTC (since no timezone was specified), but that has the special property of always being the same date when adjusted to any timezone. A fraction of a second after noon GMT/UTC would have that same property.
This comment has been removed by the author.
ReplyDeleteCarlos, since I reverted to the Python 2 version (which is working perfectly) I can't be certain which version of PS3 I was using. I believe I saved the PS3 version I was using in a new folder. Based on that, here is the last of the change log in that setup.py file:
ReplyDelete#14Dec2023*cgn
# - Update to python3
# Define the required minimum Python version. Make check compatible with Python 2 since
# a lot of users are coming from 2.7.15
import sys
REQUIRED_MAJOR = 3
REQUIRED_MINOR = 10
I had emailed Cal directly with this issue. I don't know if his response would be useful for you:
"I don't have a theory on why that behavior. Hard to imagine what Python code would cause that truncation. I would tend to suspect an overly greedy RegEx."
Ken, as luck would have it, I just recently started using Discover again and I noticed new transactions were not coming in. Behold, the same issue you experienced. FITID getting truncated! Doesn't happen with any other credit cards. Weird. Hopefully how that I have a test case I can track down what is going on! Thank you again for pointing it out.
DeleteCarlos
At least in my case, I never stopped running the Discover Card scrubber that came with the original PocketSense 2. That scrubber dates back to direct OFX downloads and Discover would do some wonky stuff with their FITIDs which Pocket Sense 2 attempted to fix. Preliminary conclusion is that that scrubber is messing up the outputs of the BoA MFP ofx files, which do not have FITID issues. I'll have to go track down why the behavior is different in PocketSense3 and PocketSense2, but I think easiest solution will be to eliminate the scrubber altogether, as it serves no other purpose.
DeleteI wasn't subscribed to this continuation so I missed your replies. Guess I should have checked before I did a little detective work and came to the same conclusion. Cal had asked me to negate the scrubbers as a possible cause and after finding out getdata doesn't work if scrubber.py isn't active, I determined that scrub_CL7.py isn't a problem and scrub_discover.py causes truncation. Since I'm not a Python programmer I'm going to leave it to you and Cal to figure out why scrub_discover.py works with PS2 and truncates with PS3. I'm back using your PS3 version without any issue (that I've found so far). By the way, I use Jon Larssen's BoAMFP to generate OFX files for all non-Fidelity bank and credit card accounts (including Discover; which is now Capital One!)
DeleteIt's good that the problem is solved.
Deletesubscribed
ReplyDeleteI had earlier posted an issue I was finding of the final price of the day not updating the middle of the day transactions for ETFs if there was a transaction on the last day of the downloaded statement. This seems to be because of Money getting confused with time stamps.
ReplyDeleteThe transaction and the portfolio position in the generated OFX have the same time stamp. The downloaded CSV files from Fidelity don't seem to have a timestamp, only a date. So, I believe F2O generates a timestamp for DTAS something like 20260821120000. (If this is UTC, then it seems incorrect for US time closing but that isn't the issue). The time is the same for all entries both for the transaction on the last day and the portfolio position. This is why Money is often using the transaction price as the final price. For Mutual Funds this does not matter since the last day price and last day transaction are both at the end of the day and the same. But for ETFs traded during the day this makes a difference. Don't know what the solution to this would be.
GV: With F2O, it is best to do the downloads before the market closes, or to wait until the mutual funds have been priced... which might be as late as 9PM. Before market close (including before market open) computes the close for the previous day, and uses that with the appropriate date. I tend to use the next morning or on the weekend.
ReplyDeleteSo while the market is open (until 4pm ET), the previous close is used.
The deal with dates like 20260821120000 is the 120000 represents a time of day of 12 noon. That will present the correct date regardless of time zone. With other times, Money can compute a wrong date.
Here is a quote from the OFX 1.6 OFX specification: :Note: Developers should consider the possibility of a date change due to timezone conversion. A datetime value in the GMT timezone with a time of 12:00:00 (noon)
would be converted to another time on the same date in every timezone. For example,
199812251200 remains Christmas Day in every timezone."
GV: It occurs to me that you might want F2O to bring in intraday prices. If prices come in via OFX, Money will not accept a later update for price for that day. That is the way it is, and it is the reason that F2O attempts to only bring in closing prices with the appropriate date for the close.
ReplyDeleteI would note that when Fidelity provided direct OFX, the dates could be a bit off... like having prices dated on Saturday, which never happens for real.
If you want intraday prices, I suggest MSMoneyQuotes. https://gaiersoftware.com/Money The price is a one-time $10. Very reasonable. Free trial. MSMoneyQuotes makes use of the internal quote feature that Money had had when it fetched prices on its own, rather than getting quotes via OFX. These quotes can be overwritten by later downloads.
The context of my previous post was not clear. I am NOT doing any intraday downloads. I only download on a Sunday to bring my MSM up to date and all prices will have stabilized by then. Let us keep this as the context for the issue. Forget mutual funds. Consider only ETFs.
ReplyDeleteSuppose there is an ETF transaction intraday on the Fri before the Sun download. This transaction is done at the intraday price where it occurred. Assume that this is somewhere midday, so the final price of the ETF at the end of the day is different from the transaction price. To record this transaction at the price of the transaction, the activity log entry is used. This as downloaded from Fidelity on the Sunday has a date but not a time stamp. Which means, the timestamp generated for the transaction in the OFX file has the date for that Fri and a generated timestamp of 120000 by F2O (as far as I can see). When Money sees this transaction in the import it creates that transaction and updates the price list with that price of the transaction (intraday) FOR THAT FRI DATE as a buy or sell depending on the transaction. The portfolio download on the other hand has the final price for the Fri. But since the download only has the date and not time, the same timestamp as the transaction is generated for the portfolio so both the portfolio item in OFX and the transaction item in OFX have the exact same date and time but with two different prices.
If MSM sees the transaction item first and records that midday price for the Fri, when it encounters the portfoilio position item with the same timestamp and the final price at the end of the day Fri, it seems just to drop the latter since a price for that timestamp has already been encountered. It will not update the price twice for the same day. So you land up with that ETF as having the transaction price as the last price recorded on Fri rather than its final price for the position. So the total portfolio value will not match the total in Fidelity on Sun. It as to be manually updated to the last price of the Fri.
If you generate a time stamp of 120000 for transaction and 120001 for portfolio position item, then I am thinking that Money will see the portfolio position as being at a later timestamp than transaction and update the price again. So, you will have the correct portfolio totals on Sun to start the next week.
GV: That is an interesting and clear idea.
ReplyDeleteMy experiments with existing files show that if there is a Buy or Sell for a date, Money ignores the downloaded price for that date.
You can find the price history by going to the investment details for an investment.
UpdatePrices->UpdatePricesManually Look through the table. You will see that if there is a Buy or a Sell, there is no Update or Oneline price listed for that date. The price shown is for the latest-entered Buy or Sell for that day.
In OFX, Money uses the first <DTASOF> as the date of quotes.
While I doubt making the change will matter, I may try that at some point.
If you want to try it out, you can just change one line.
Line 1291 of the latest F2O.py contains
dtasof_ele=("<DTASOF>"+dt_Ymd+"120000\n") #coulf move \n
You could change that line, using Notepad, to be this:
dtasof_ele=("<DTASOF>"+dt_Ymd+"120001\n") #modified as test
Note that time 120000 is noon GMT/UTC (since no timezone was specified), but that has the special property of always being the same date when adjusted to any timezone. A fraction of a second after noon GMT/UTC would have that same property.
GV: I tried. I changed F2O.py. I actually did a Buy today. Result was that the price that Money took as the end-of-day price was the price of the Buy.
ReplyDeleteIn the OFX file, the DTASOF dates were all 20260908120200. The DTTRADE was 20260908120000.
It was worth a try, but I think that this confirms what I suspected.