Don't listen to that voice. God will take you right where you're at and lead you. He'll take your wanderings and somehow fashion your future into working with those wanderings so that your wanderings will have great meaning.
I graduated from college in December of 1990. I didn't know what to do. Colleges these days make sure students are guided toward employment. Maybe they did that back then too but if so I slipped through the cracks. I decided to go to New Jersey and be a nanny. That was a disaster. So I came home four months later and worked a couple of jobs. And one day decided to join the Army. I am not Army material. I never was, but I didn't know what else to do. I even knew it wasn't God leading me. He tried to stop me. It felt like to me He was shaking His head at my decision. But He gave me a good job as a journalist and photographer and used my time in the Army to do some really neat things. I even met my husband in Africa. Somehow God fashioned my days so that the Army fit for me in a weird sort of way.
If I hadn't chosen to do this I don't know where God would have led me, but it doesn't matter for God used me where I was at and somehow made my wanderings link up like lego blocks with His plan. I don't know how God can do that but He can. He will do that for anyone who seeks Him.
These days I know I am where I am supposed to be. You can have that assurance too. Seek God and He will give you His peace. He will guide you. He will lead you.
"And we know that God causes all things to work together for good to those who love God, to those who are called according to His purpose." Romans 8:28