Carlos, since I reverted to the Python 2 version (which is working perfectly) I can't be certain which version of PS3 I was using. I believe I saved the PS3 version I was using in a new folder. Based on that, here is the last of the change log in that setup.py file: #14Dec2023*cgn # - Update to python3
# Define the required minimum Python version. Make check compatible with Python 2 since # a lot of users are coming from 2.7.15 import sys REQUIRED_MAJOR = 3 REQUIRED_MINOR = 10
I had emailed Cal directly with this issue. I don't know if his response would be useful for you: "I don't have a theory on why that behavior. Hard to imagine what Python code would cause that truncation. I would tend to suspect an overly greedy RegEx."
Ken, as luck would have it, I just recently started using Discover again and I noticed new transactions were not coming in. Behold, the same issue you experienced. FITID getting truncated! Doesn't happen with any other credit cards. Weird. Hopefully how that I have a test case I can track down what is going on! Thank you again for pointing it out.
At least in my case, I never stopped running the Discover Card scrubber that came with the original PocketSense 2. That scrubber dates back to direct OFX downloads and Discover would do some wonky stuff with their FITIDs which Pocket Sense 2 attempted to fix. Preliminary conclusion is that that scrubber is messing up the outputs of the BoA MFP ofx files, which do not have FITID issues. I'll have to go track down why the behavior is different in PocketSense3 and PocketSense2, but I think easiest solution will be to eliminate the scrubber altogether, as it serves no other purpose.
I wasn't subscribed to this continuation so I missed your replies. Guess I should have checked before I did a little detective work and came to the same conclusion. Cal had asked me to negate the scrubbers as a possible cause and after finding out getdata doesn't work if scrubber.py isn't active, I determined that scrub_CL7.py isn't a problem and scrub_discover.py causes truncation. Since I'm not a Python programmer I'm going to leave it to you and Cal to figure out why scrub_discover.py works with PS2 and truncates with PS3. I'm back using your PS3 version without any issue (that I've found so far). By the way, I use Jon Larssen's BoAMFP to generate OFX files for all non-Fidelity bank and credit card accounts (including Discover; which is now Capital One!)
I had earlier posted an issue I was finding of the final price of the day not updating the middle of the day transactions for ETFs if there was a transaction on the last day of the downloaded statement. This seems to be because of Money getting confused with time stamps.
The transaction and the portfolio position in the generated OFX have the same time stamp. The downloaded CSV files from Fidelity don't seem to have a timestamp, only a date. So, I believe F2O generates a timestamp for DTAS something like 20260821120000. (If this is UTC, then it seems incorrect for US time closing but that isn't the issue). The time is the same for all entries both for the transaction on the last day and the portfolio position. This is why Money is often using the transaction price as the final price. For Mutual Funds this does not matter since the last day price and last day transaction are both at the end of the day and the same. But for ETFs traded during the day this makes a difference. Don't know what the solution to this would be.
GV: With F2O, it is best to do the downloads before the market closes, or to wait until the mutual funds have been priced... which might be as late as 9PM. Before market close (including before market open) computes the close for the previous day, and uses that with the appropriate date. I tend to use the next morning or on the weekend.
So while the market is open (until 4pm ET), the previous close is used.
The deal with dates like 20260821120000 is the 120000 represents a time of day of 12 noon. That will present the correct date regardless of time zone. With other times, Money can compute a wrong date.
Here is a quote from the OFX 1.6 OFX specification: :Note: Developers should consider the possibility of a date change due to timezone conversion. A datetime value in the GMT timezone with a time of 12:00:00 (noon) would be converted to another time on the same date in every timezone. For example, 199812251200 remains Christmas Day in every timezone."
GV: It occurs to me that you might want F2O to bring in intraday prices. If prices come in via OFX, Money will not accept a later update for price for that day. That is the way it is, and it is the reason that F2O attempts to only bring in closing prices with the appropriate date for the close.
I would note that when Fidelity provided direct OFX, the dates could be a bit off... like having prices dated on Saturday, which never happens for real.
If you want intraday prices, I suggest MSMoneyQuotes. https://gaiersoftware.com/Money The price is a one-time $10. Very reasonable. Free trial. MSMoneyQuotes makes use of the internal quote feature that Money had had when it fetched prices on its own, rather than getting quotes via OFX. These quotes can be overwritten by later downloads.
This comment has been removed by the author.
ReplyDeleteCarlos, since I reverted to the Python 2 version (which is working perfectly) I can't be certain which version of PS3 I was using. I believe I saved the PS3 version I was using in a new folder. Based on that, here is the last of the change log in that setup.py file:
ReplyDelete#14Dec2023*cgn
# - Update to python3
# Define the required minimum Python version. Make check compatible with Python 2 since
# a lot of users are coming from 2.7.15
import sys
REQUIRED_MAJOR = 3
REQUIRED_MINOR = 10
I had emailed Cal directly with this issue. I don't know if his response would be useful for you:
"I don't have a theory on why that behavior. Hard to imagine what Python code would cause that truncation. I would tend to suspect an overly greedy RegEx."
Ken, as luck would have it, I just recently started using Discover again and I noticed new transactions were not coming in. Behold, the same issue you experienced. FITID getting truncated! Doesn't happen with any other credit cards. Weird. Hopefully how that I have a test case I can track down what is going on! Thank you again for pointing it out.
DeleteCarlos
At least in my case, I never stopped running the Discover Card scrubber that came with the original PocketSense 2. That scrubber dates back to direct OFX downloads and Discover would do some wonky stuff with their FITIDs which Pocket Sense 2 attempted to fix. Preliminary conclusion is that that scrubber is messing up the outputs of the BoA MFP ofx files, which do not have FITID issues. I'll have to go track down why the behavior is different in PocketSense3 and PocketSense2, but I think easiest solution will be to eliminate the scrubber altogether, as it serves no other purpose.
DeleteI wasn't subscribed to this continuation so I missed your replies. Guess I should have checked before I did a little detective work and came to the same conclusion. Cal had asked me to negate the scrubbers as a possible cause and after finding out getdata doesn't work if scrubber.py isn't active, I determined that scrub_CL7.py isn't a problem and scrub_discover.py causes truncation. Since I'm not a Python programmer I'm going to leave it to you and Cal to figure out why scrub_discover.py works with PS2 and truncates with PS3. I'm back using your PS3 version without any issue (that I've found so far). By the way, I use Jon Larssen's BoAMFP to generate OFX files for all non-Fidelity bank and credit card accounts (including Discover; which is now Capital One!)
DeleteIt's good that the problem is solved.
Deletesubscribed
ReplyDeleteI had earlier posted an issue I was finding of the final price of the day not updating the middle of the day transactions for ETFs if there was a transaction on the last day of the downloaded statement. This seems to be because of Money getting confused with time stamps.
ReplyDeleteThe transaction and the portfolio position in the generated OFX have the same time stamp. The downloaded CSV files from Fidelity don't seem to have a timestamp, only a date. So, I believe F2O generates a timestamp for DTAS something like 20260821120000. (If this is UTC, then it seems incorrect for US time closing but that isn't the issue). The time is the same for all entries both for the transaction on the last day and the portfolio position. This is why Money is often using the transaction price as the final price. For Mutual Funds this does not matter since the last day price and last day transaction are both at the end of the day and the same. But for ETFs traded during the day this makes a difference. Don't know what the solution to this would be.
GV: With F2O, it is best to do the downloads before the market closes, or to wait until the mutual funds have been priced... which might be as late as 9PM. Before market close (including before market open) computes the close for the previous day, and uses that with the appropriate date. I tend to use the next morning or on the weekend.
ReplyDeleteSo while the market is open (until 4pm ET), the previous close is used.
The deal with dates like 20260821120000 is the 120000 represents a time of day of 12 noon. That will present the correct date regardless of time zone. With other times, Money can compute a wrong date.
Here is a quote from the OFX 1.6 OFX specification: :Note: Developers should consider the possibility of a date change due to timezone conversion. A datetime value in the GMT timezone with a time of 12:00:00 (noon)
would be converted to another time on the same date in every timezone. For example,
199812251200 remains Christmas Day in every timezone."
GV: It occurs to me that you might want F2O to bring in intraday prices. If prices come in via OFX, Money will not accept a later update for price for that day. That is the way it is, and it is the reason that F2O attempts to only bring in closing prices with the appropriate date for the close.
ReplyDeleteI would note that when Fidelity provided direct OFX, the dates could be a bit off... like having prices dated on Saturday, which never happens for real.
If you want intraday prices, I suggest MSMoneyQuotes. https://gaiersoftware.com/Money The price is a one-time $10. Very reasonable. Free trial. MSMoneyQuotes makes use of the internal quote feature that Money had had when it fetched prices on its own, rather than getting quotes via OFX. These quotes can be overwritten by later downloads.